United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. 



I0/732.7KI 



FILING DATE 



12/10/20(0 



46073 7590 12/05/2008 

IBM CORPORATION (VE) 
C/O VOLEL EMILE 
P. O. BOX 162485 
AUSTIN, TX 78716 



FIRST NAMED INVENTOR 



Michael Wayne Brown 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



AUS920030341US1 



FITZPATRICK, ATIBA O 



PAPER NUMBER 



DELIVERY MODE 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/732,781 


Applicant(s) 

BROWN ET AL. 


Examiner 

ATIBA O. FITZPATRICK 


Art Unit 

2624 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )KI Responsive to communication(s) filed on 29 July 2008 . 
2a )^ This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) |EI Claim(s) 1-7,9-15,17-23 and 25-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) 13 Claim(s) 1-7.9-15.17-23 and 25-30 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) D Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 
Paper No(s)/Mail Date 05/07/2008 . 6) □ Other: . 



PTOL-T26 d (Rev e 08-06r 



Office Action Summary 



Part of Paper No./Mail Date 20081 105 



Application/Control Number: 10/732,781 
Art Unit: 2624 



Page 2 



DETAILED ACTION 

Response to Arguments 

Objections to the title and abstract are withdrawn in light of amendments. 

Applicant's arguments have been considered but are moot in view of the new ground(s) 
of rejection necessitated by amendment. In response to Applicant's arguments for the 
prior statutory double patenting rejections, claims are now rejected under nonstatutory 
(obvious-type) double patenting as indicated below. 

Claim Rejections - 35 USC § 101 

35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

Claims 9-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed 
to non-statutory subject matter. Claim 9 includes the limitation "storage medium having 
instructions", but this does not require that the instructions are stored on the medium. 
Note that the claim must explicitly state that the instructions or program is stored on the 
computer readable medium. 

Claim Rejections - 35 USC §112 

The following is a quotation of the second paragraph of 35 U.S.C. 112: 
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The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

Claims 1-7, 9-15, 17-23, and 25-30 are rejected under 35 U.S.C. 112, second 
paragraph, as being indefinite for failing to particularly point out and distinctly claim the 
subject matter which applicant regards as the invention. Claims 1,9, 17, and 25 include 
the limitation "to improve a speaker's speech making ability". It is unclear how this 
system can improve a speaker's ability. How can this system improve a person's ability? 
If the system provides feedback to the speaker from the listeners, it is still up to the 
speaker to utilize the feedback to improve his/her own ability. Thus, in this case, the 
system allows the speaker to improve his/her own ability. The only other interpretation 
that the office can perceive that would suffice would be if this was some sort of 
biomedical device that directly alters a speaker's voice or speech characteristics (such 
as volume). Still, in this case, one might argue that the speaker's ability is not improved 
by the system, but instead, the system would compensate for the speaker's ability. Also, 
note that the excerpt is understood to be intended use or purpose because there is no 
positive recitation that necessitates that the following is required to be performed: MPEP 
2106. Therefore, the phrase does not receive patentable weight. 

Also, claim 9 includes the limitation "storage medium having instructions", but this does 
not require that the instructions are stored on the medium. However, it is not clear what 
is meant by "having". Explicitly stating that the instructions are stored on the medium 
would render the limitation definite. 
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Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1, 4, 9, 12, 17, 20, 25, and 28 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over USPGPubN 200201 01 505 (Gutta) in view of USPN 71 1 71 57 
(Taylor) and USPGPubN 20020116197 (Erten). 

As per claim 1 , Gutta teaches a method of using stored audio and video data recorded 
at a conference as a speech analysis tool comprising (Limitations present only within 
the preamble are not given patentable weight): 

the facial expression being exhibited by one or more of the participants listening to 
speech at the conference; determining, using the stored video data in conjunction with 
an automated facial decoding system, whether at least one participant listening to the 
speech exhibited the indicated facial expression (Gutta: abstract; Figs. 1-4; paras 8- 
10, 17: The acoustic cues and expressions of a listener are detected to determine 
that the listener should now receive the focus); and 

analyzing, in response to determining that the at least one participant listening to the 
speech exhibited the facial expression, the speech by analyzing the stored video data 
representing the at least one participant exhibiting the facial expression and stored 
audio data representing what was being said in the speech when the at least one 
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participant exhibited the facial expression (Gutta: abstract; Figs. 1-5; paras 8-10, 17, 
36-38, 59-75: The acoustic cues and expressions of a listener are detected to 
determine that the listener should now receive the focus) to (Note that the 
following is understood to be intended use or purpose because there is no 
positive recitation that necessitates that the following is required to be 
performed: MPEP 2106. Therefore, the following phrase does not receive 
patentable weight) improve a speaker's speech making ability. 

Gutta does not teach indicating a facial expression for which to search from the stored 
video data; to improve a speaker's speech making ability. 

Taylor teaches indicating a facial expression for which to search from the stored video 
data (Taylor: Fig. 3: s12-20: col 5, line 63- col 6, line 34; Fig. 23: S302-320). 

Thus, it would have been obvious for one of ordinary skill in the art at the time the 
invention was made to implement the teachings of Taylor into Gutta since Gutta 
suggests a system for automatically focusing the camera on the speaker on a 
videoconference using expressions of the speakers and other participants for making 
the determination in general and Taylor suggests the beneficial use of a system for 
automatically focusing the camera on the speaker on a videoconference using 
expressions of the speakers and other participants for making the determination 
wherein a user specifies the particular expressions that cue for the camera focus to 
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change to another participant in the analogous art of image processing. It would have 
been obvious for one of ordinary skill in the art at the time the invention was made to 
allow the user to specify particular expression cues in an initialization stage in order to 
allow for flexibility and customization considering that an operator may know how best 
to customize the system for difference settings and groups/types of participants. 
Furthermore, one of ordinary skill in the art at the time the invention was made could 
have combined the elements as claimed by known methods and, in combination, each 
component functions the same as it does separately. One of ordinary skill in the art at 
the time the invention was made would have recognized that the results of the 
combination would be predictable. 

Erten teaches analyzing, in response to determining that the at least one participant 
listening to the speech exhibited the facial expression, the speech by analyzing the 
stored video data representing the at least one participant exhibiting the facial 
expression and stored audio data representing what was being said in the speech when 
the at least one participant exhibited the facial expression to improve a speaker's 
speech making ability (Erten: abstract; Fig. 1-3 and 14-16; paras 3 and 7-22). 

Thus, it would have been obvious for one of ordinary skill in the art at the time the 
invention was made to implement the teachings of Erten into Gutta since Gutta 
suggests a system that recognizes and processes expression and speech information in 
general and Erten suggests the beneficial use of a system that recognizes and 
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processes expression and speech information in order to enhance the speaker's speech 
since "Unfortunately, speech often occurs in the presence of noise. This noise may take 
many forms such as natural sounds, machinery, music, speech from other people, and 
the like. Traditionally, such noise is reduced through the use of acoustic filters. While 
such filters are effective, they are frequently not adequate in reducing the noise content 
in a speech signal to an acceptable level" (Erten: para 5) in the analogous art of image 
processing. Furthermore, one of ordinary skill in the art at the time the invention was 
made could have combined the elements as claimed by known methods and, in 
combination, each component functions the same as it does separately. One of ordinary 
skill in the art at the time the invention was made would have recognized that the results 
of the combination would be predictable. 

As per claim 4, Gutta in view of Taylor and Erten teaches the method of Claim 1 
wherein the analyzing step includes the step of analyzing the stored video data 
representing the at least one participant exhibiting the facial expression and stored 
audio data representing what was being said in the speech when the at least one 
participant exhibited the facial expression (Gutta: abstract; Figs. 1-5; paras 8-10, 17, 
36-38, 59-75). 

Gutta in view of Erten does not teach includes displaying a percentage of participants 
who exhibited the expression. 
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Taylor teaches includes displaying a percentage of participants who exhibited the 
expression (Taylor: Fig. 16a-b; Fig. 33a-b: The expression being exhibited is the 
physical head angle and direction of view as observed by the camera). 

Thus, it would have been obvious for one of ordinary skill in the art at the time the 
invention was made to implement the teachings of Taylor into Gutta since Gutta 
suggests a audio visual speech processing application in general and Taylor suggests 
the beneficial use of displaying the proportion of participants exhibiting an expression 
(directional glance) as to provide an indication of the proportion of time that a participant 
exhibited a certain gaze in the analogous art of speech and video processing. 
Furthermore, one of ordinary skill in the art at the time the invention was made could 
have combined the elements as claimed by known methods and, in combination, each 
component functions the same as it does separately. One of ordinary skill in the art at 
the time the invention was made would have recognized that the results of the 
combination would be predictable. 

Arguments made in rejecting claims 9, 17, and 25 are analogous to arguments for 
rejecting claim 1 . Note that limitations present only in the preamble are not given 
patentable weight. 

Arguments used to reject claims 12, 20, and 28 are analogous to arguments used to 
reject claim 4. 
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Claims 2, 3, 5, 10, 11, 13, 18, 19, 21 , 26, 27, and 29 are rejected under 35 
U.S.C. 103(a) as being unpatentable over USPGPubN 20020101505 (Gutta) in view of 
USPN 71 1 71 57 (Taylor) and USPGPubN 200201 1 61 97 (Erten) as applied to claims 1 , 
9, 17, and 25 respectively, and further in view of USPN 6585521 (Obrador). 

As per claim 2, Gutta in view of Taylor and Erten teaches the method of Claim 1 
wherein analyzing the stored video data representing the at least one participant 
exhibiting the facial expression and stored audio data representing what was being said 
in the speech when the at least one participant exhibited the facial expression (Gutta: 
abstract; Figs. 1-5; paras 8-10, 17, 36-38, 59-75). 

Gutta in view of Taylor and Erten does not teach includes charting expressions 
exhibited by the at least one participant over time. 

Obrador teaches includes charting expressions exhibited by the at least one participant 
overtime (Obrador: Fig. 4a: 480; Figs. 4-6). 

Thus, it would have been obvious for one of ordinary skill in the art at the time the 
invention was made to implement the teachings of Obrador into Gutta since Gutta 
suggests a audio visual speech processing application in general and Obrador suggests 
the beneficial use of charting for speech video indexing based on user emotional 
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feedback as to "Video indexing allows a user to have easy access to different sections 
of the video sequence" (Obrador: col 1, line 21) in the analogous art of speech and 
video processing. Furthermore, one of ordinary skill in the art at the time the invention 
was made could have combined the elements as claimed by known methods and, in 
combination, each component functions the same as it does separately. One of ordinary 
skill in the art at the time the invention was made would have recognized that the results 
of the combination would be predictable. 



As per claim 3, Gutta in view of Taylor and Erten teaches the method of Claim 1 
wherein the analyzing step includes the step of analyzing the stored video data 
representing the at least one participant exhibiting the facial expression and stored 
audio data representing what was being said in the speech when the at least one 
participant exhibited the facial expression (Gutta: abstract; Figs. 1-5; paras 8-10, 17, 
36-38, 59-75). 

Gutta in view of Taylor and Erten does not teach includes displaying a percentage of 
time the expression is exhibited by the at least one participant. 

Obrador teaches includes displaying a percentage of time the expression is exhibited by 
the at least one participant (Obrador: Fig. 4a: 430, 440; Figs. 4-6). 
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As per claim 5, Gutta in view of Taylor and Erten teaches the method of Claim 1 
wherein the analyzing step includes the step of analyzing the stored video data 
representing the at least one participant exhibiting the facial expression and stored 
audio data representing what was being said in the speech when the at least one 
participant exhibited the facial expression (Gutta: abstract; Figs. 1-5; paras 8-10, 17, 
36-38, 59-75). 

Gutta in view of Taylor and Erten does not teach includes displaying a percentage of 
time participants exhibited an expression. 

Obrador teaches includes displaying a percentage of time participants exhibited an 
expression (Obrador: Fig. 4a: 430, 440; Figs. 4-6). 

Arguments made in rejecting claims 10, 18, and 26 are analogous to arguments for 
rejecting claim 2. 

Arguments made in rejecting claims 11, 19, and 27 are analogous to arguments for 
rejecting claim 3. 

Arguments made in rejecting claims 13, 21, and 29 are analogous to arguments for 
rejecting claim 5. 
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Claims 6, 7, 14, 15, 22, 23, and 26 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over USPGPubN 200201 01 505 (Gutta) in view of USPN 71 1 71 57 (Taylor) 
and USPGPubN 200201 1 61 97 (Erten) as applied to claims 1 , 9, 1 7, and 25 above, and 
further in view of USPN 6404438 (Hatlelid). 

As per claim 6, Gutta in view of Taylor and Erten teaches the method of Claim 1 
wherein the step of determining whether at least one participant listening to the speech 
exhibited the indicated facial expression (Gutta: abstract; Figs. 1-5; paras 8-10, 17, 
36-38, 59-75) includes the step of passing the data through a filter for filtering out the 
expression (Gutta: Figs. 3-5; para 10 34, 40-58). 

Gutta in view of Taylor and Erten does not teach passing the data through a cultural 
filter for filtering out the expression if it is a culturally based expression. 

Hatlelid teaches passing the data through a cultural filter for filtering out the expression 
if it is a culturally based expression (Fig. 7: the phrases in 804 pertain to cultures: 
col 11, line 13 -col 12, line 67; Also Figs. 3-6). 

Thus, it would have been obvious for one of ordinary skill in the art at the time the 
invention was made to implement the teachings of Hatlelid into Gutta since Gutta 
suggests a communication environment that assesses users/speakers expressions and 
uses the determinations to enhance communication in general and Hatlelid suggests 
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the beneficial use of a communication environment that assesses users/speakers 
expressions and uses the determinations to enhance communication wherein individual 
and cultural personality filters are used as to "reflect generally the user's own behavior 
and personality... a user can select from a variety of personality types to represent the 
user, including 'hip-hop,' 'upper-class,' 'rocker', or the like" (Hatlelid: coM , lines 19-52) 
in the analogous art of image processing. Furthermore, one of ordinary skill in the art at 
the time the invention was made could have combined the elements as claimed by 
known methods and, in combination, each component functions the same as it does 
separately. One of ordinary skill in the art at the time the invention was made would 
have recognized that the results of the combination would be predictable. 

Arguments made in rejecting claims 14, 22, and 30 are analogous to arguments for 
rejecting claim 6. 

As per claim 7, Gutta in view of Taylor and Erten teaches the method of Claim 1 
wherein the step of determining whether at least one participant listeninq to the speech 
exhibited the indicated facial expression (Gutta: abstract; Figs. 1-5; paras 8-10, 17, 
36-38, 59-75) includes the step of passing the data through an individual filter for 
filtering out the expression if it is an expression particular to the participant (Gutta: Figs. 
3-5; para 10 34, 40-58). 



Application/Control Number: 10/732,781 Page 14 

Art Unit: 2624 

Hatlelid also teaches passing the data through an individual filter for filtering out the 
expression if it is an expression particular to the participant (Fig. 7: col 1 1 , line 1 3 - col 
12, line 67; Also Figs. 3-6 and 11-14). 



Arguments made in rejecting claims 15 and 23 are analogous to arguments for rejecting 
claim 7. 



Double Patenting 

The nonstatutory double patenting rejection is based on a judicially created 
doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the 
unjustified or improper timewise extension of the "right to exclude" granted by a patent 
and to prevent possible harassment by multiple assignees. A nonstatutory 
obviousness-type double patenting rejection is appropriate where the conflicting claims 
are not identical, but at least one examined application claim is not patentably distinct 
from the reference claim(s) because the examined application claim is either anticipated 
by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 
F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 
USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 
1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 
F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 
USPQ 644 (CCPA 1969). 

A timely filed terminal disclaimer in compliance with 37 CFR 1 .321 (c) or 1 .321 (d) 
may be used to overcome an actual or provisional rejection based on a nonstatutory 
double patenting ground provided the conflicting application or patent either is shown to 
be commonly owned with this application, or claims an invention made as a result of 
activities undertaken within the scope of a joint research agreement. 

Effective January 1 , 1994, a registered attorney or agent of record may sign a 
terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 
37 CFR 3.73(b). 



Claims 1,9, 17, and 25 are rejected on the ground of nonstatutory obviousness-type 
double patenting as being unpatentable over claims 1, 8, 15, and 22 of U.S. Patent No. 
20050131744 in view of USPGPubN 20020101505 (Gutta), USPN 7117157 (Taylor), 
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and USPGPubN 200201 16197 (Erten). The scope of both sets of claims from both 
applications is essentially identical. Where Applicant deems that the scope of the sets of 
claims from the separate applications differs, Applicant is referred to the secondary 
references as applied (cited) in the 35 USC 103 rejection section above. 

Claims 6, 14, 22, and 30 are rejected on the ground of nonstatutory obviousness-type 
double patenting as being unpatentable over claims 2, 9, 16, and 23 of U.S. Patent No. 
20050131744 in view of USPGPubN 20020101505 (Gutta), USPN 7117157 (Taylor), 
USPGPubN 20020116197 (Erten), and USPN 6404438 (Hatlelid). The scope of both 
sets of claims from both applications is essentially identical. Where Applicant deems 
that the scope of the sets of claims from the separate applications differs, Applicant is 
referred to the secondary references as applied (cited) in the 35 USC 103 rejection 
section above. 

Claims 7, 15, and 23 are rejected on the ground of nonstatutory obviousness-type 
double patenting as being unpatentable over claims 3, 10, 17, and 24 of U.S. Patent No. 
20050131744 in view of USPGPubN 20020101505 (Gutta), USPN 7117157 (Taylor), 
USPGPubN 20020116197 (Erten), and USPN 6404438 (Hatlelid). The scope of both 
sets of claims from both applications is essentially identical. Where Applicant deems 
that the scope of the sets of claims from the separate applications differs, Applicant is 
referred to the secondary references as applied (cited) in the 35 USC 103 rejection 
section above. 
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Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy 
as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Atiba Fitzpatrick whose telephone number is (571) 270- 
5255. The examiner can normally be reached on M-F 10:00am-6pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Samir Ahmed can be reached on (571)272-7413. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published 
applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO 
Customer Service Representative or access to the automated information system, call 
800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Atiba Fitzpatrick 
Patent Examiner 
/Samir A. Ahmed/ 

Supervisory Patent Examiner, Art Unit 2624 



